Skip to content

DOC: Correct citations in doc/references.bib (audit pass)#1747

Merged
romanlutz merged 3 commits into
microsoft:mainfrom
romanlutz:romanlutz/ref-check-bibtex
May 18, 2026
Merged

DOC: Correct citations in doc/references.bib (audit pass)#1747
romanlutz merged 3 commits into
microsoft:mainfrom
romanlutz:romanlutz/ref-check-bibtex

Conversation

@romanlutz

@romanlutz romanlutz commented May 18, 2026

Copy link
Copy Markdown
Contributor

Description

Ran academic-refchecker against doc/references.bib and audited every arXiv-linked entry against arXiv metadata to catch citations where the cited title, authors, or year did not match the actual paper. The original pass flagged 15 errors / 3 hallucinations; a follow-up manual audit caught three more issues that automated checking missed.

Fixes to doc/references.bib

Errors corrected (wrong title/authors/IDs, malformed fields, encoding artifacts):

  • promptfoo2025ccp: actual blog post title is "1,156 Questions Censored by DeepSeek" (the previous title was paraphrased from the post body); set author to byline (Ian Webster), added publication month, repaired prior mojibake.
  • bullwinkel2024airtlessons -> bullwinkel2025airtlessons: arXiv submission date is 2025-01-13; the year field was already 2025 but the citekey lagged.
  • ji2023beavertails: added missing co-author Boyuan Chen. The arXiv metadata for 2307.04657 lists "Chi Zhang" twice in his slot; the NeurIPS 2023 final author list and the paper's email block confirm Chen.
  • scheuerman2025transphobia: moved ACM TOCHI venue into journal, added doi.
  • aqrawi2024singleturncrescendo, bethany2024mathprompt, andriushchenko2024tense, mckee2024transparency, and several others: title/author/year/url corrections (see commit history for the full per-entry list).

Citekey renames (and propagation to all callers):

  • chu2023harmfulqa -> bhardwaj2023harmfulqa (the paper PyRIT actually loads is Bhardwaj and Poria, arXiv:2308.09662 - the previous entry pointed to an unrelated paper despite the same dataset name).
  • li2024flipattack -> liu2024flipattack (first author is Yue Liu, not Li).
  • yu2025comicjailbreak -> tan2026comicjailbreak (first author is Yixuan Tan; arXiv ID/year corrected).

Caller updates for the citekey renames

  • doc/bibliography.md
  • doc/code/datasets/1_loading_datasets.{py,ipynb}
  • doc/code/executor/attack/flip_attack.{py,ipynb}
  • doc/code/targets/0_prompt_targets.md
  • pyrit/datasets/seed_datasets/remote/harmful_qa_dataset.py
  • pyrit/datasets/seed_datasets/remote/comic_jailbreak_dataset.py
  • pyrit/executor/attack/single_turn/flip_attack.py

Verification

Re-ran academic-refchecker after the first commit: 15 -> 4 errors, 3 -> 1 hallucinated. The remaining flags are either tool false-positives (e.g. pybtex/refchecker drops the single-name first author of aakanksha2024multilingual regardless of brace or comma form) or unverifiable industry blog posts. The follow-up manual audit then confirmed every arXiv URL in the bib now points to the correctly-cited paper.

Tests and Documentation

Documentation-only change. All pre-commit hooks pass (ruff, ty, link-check, nbstripout, documentation-structure validation). The citekey renames were propagated to every .py, .ipynb, and .md caller in the repo; grep for each old key returns no results.

romanlutz and others added 2 commits May 18, 2026 05:26
Ran refchecker (https://github.com/markrussinovich/refchecker) on
doc/references.bib and corrected the entries it flagged as having wrong
authors, titles, arXiv IDs, or other metadata. Reduced errors from 15 to
4 and hallucination flags from 3 to 1 (remaining flags are tool false
positives on web-page citations and the TDC23 competition entry).

Hallucinated/wrong-arXiv-ID fixes:
- chu2023harmfulqa -> bhardwaj2023harmfulqa: arXiv 2310.18469 is a
  gaze-estimation paper. PyRIT loads the declare-lab/HarmfulQA dataset,
  introduced in Bhardwaj & Poria, "Red-Teaming Large Language Models
  using Chain of Utterances for Safety-Alignment" (arXiv:2308.09662).
- li2024flipattack -> liu2024flipattack: arXiv 2410.02832 is by Yue Liu
  et al., not Yue Li et al.
- yu2025comicjailbreak -> tan2026comicjailbreak: arXiv 2603.21697 is
  "Structured Visual Narratives Undermine Safety Alignment in MLLMs" by
  Rui Yang Tan, Yujia Hu, Roy Ka-Wei Lee (2026), which introduces the
  ComicJailbreak benchmark.
- hines2024spotlighting: correct author list is Hines, Lopez, Hall,
  Zarfati, Zunger, Kiciman (was wrong list of 9 names).

Wrong-title fixes (arXiv ID was correct):
- aqrawi2024singleturncrescendo: "Well, that escalated quickly: The
  Single-Turn Crescendo Attack (STCA)".
- bethany2024mathprompt: "Jailbreaking Large Language Models with
  Symbolic Mathematics".
- jiang2025sosbench: "SoSBench: Benchmarking Safety Alignment on Six
  Scientific Domains"; add missing co-author Zixin Rao.

Other metadata fixes:
- promptfoo2025ccp: repair mojibake in title (Censors?rompts ->
  Censors Prompts) and brace the corporate author.
- scheuerman2025transphobia: move the journal name into a journal field
  and add the doi field.
- zou2023gcg: add missing co-authors Nicholas Carlini and Milad Nasr.
- bullwinkel2024airtlessons: year 2024 -> 2025 (arXiv submission year).
- ji2023beavertails: remove duplicate "Chi Zhang" from author list.
- aakanksha2024multilingual: add missing first author Aakanksha (use
  BibTeX comma form so pybtex recognizes a single-name author).

Citekey renames are propagated to all callers in doc/code and pyrit/.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- promptfoo2025ccp: use the actual blog post title ("1,156 Questions
  Censored by DeepSeek") and byline author (Ian Webster); add publication
  month. The previous title was paraphrased from the post body.
- bullwinkel2024airtlessons -> bullwinkel2025airtlessons: align the citekey
  with the actual arXiv submission date (2025-01-13); the year field was
  already 2025. Updated the lone caller in doc/bibliography.md.
- ji2023beavertails: add missing co-author Boyuan Chen. The arXiv metadata
  for 2307.04657 has a duplicated "Chi Zhang" entry in his slot; the NeurIPS
  2023 final author list and the paper's email block confirm Boyuan Chen.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread doc/references.bib Outdated
@hannahwestra25 hannahwestra25 self-assigned this May 18, 2026
pybtex parses the bare single-name form correctly; the trailing comma is unnecessary and renders awkwardly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@romanlutz
romanlutz added this pull request to the merge queue May 18, 2026
Merged via the queue into microsoft:main with commit 3cfa661 May 18, 2026
48 checks passed
@romanlutz
romanlutz deleted the romanlutz/ref-check-bibtex branch May 18, 2026 21:51
romanlutz added a commit to romanlutz/PyRIT that referenced this pull request May 19, 2026
Resolves conflict in doc/bibliography.md hidden-citations list against microsoft#1747 (DOC: Correct citations) by keeping main's citation-key renames and re-adding @shaikh2022second in alphabetical order.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants